1. Field of the Invention
This invention relates to log data and more particularly relates to condensing checkpoint log data.
2. Description of the Related Art
Computer software generally includes a log feature that may be used during development or during normal operation of a software application. The log feature causes the software application to report various types of information regarding the health or status of each software sub-system, statistics from system control blocks, and other highly detailed information known herein as log data. Generally, log data is analyzed by software engineers or system administrators to facilitate resolving software bugs and/or inefficiencies in the software application. Typically, log data can be produced at various levels of granularity. The different levels of granularity facilitate tracking down software errors.
However, a high granularity also produces very large quantities of log data. For each software event logged, a log entry is typically generated. The log entry is typically relatively small and provides information about the operation being performed as well as context information such as inputs, outputs, and other state information.
Log data is typically stored for subsequent analysis after the software application is executed to generate the software error. Because log data may be collected during high workload periods for the computer system and/or software application, it is desirable that the logging operation add minimal overhead to the workload. Consequently, the frequently-generated log entries are typically combined into larger groups of log entries, known herein as checkpoint log records. The checkpoint log records often include a header that identifies the number and length of log entries contained therein as well as other context information such as a timestamp. Checkpoint log records can be over one hundred times larger than individual log entries. Storing fewer large checkpoint log records requires less I/O than storing many small individual log entries.
Log data can be collected during a single execution or over a period of time in order to identify more latent software bugs. Consequently, the size of the log data may grow dramatically. Analyzing such high quantities of log data has been difficult for programmers and system administrators. With the complexities of modem software and the high quantities of log data, the debugging task becomes the proverbial search for a needle in a haystack.
Storing checkpoint log records optimizes writing to the storage devices, but makes reviewing and analysis extremely difficult. In particular, search utilities currently available such as DFSERA10 provided with the Information Management System (IMS) from IBM of Armonk, N.Y., do not permit searching for a data value within individual log entries. Instead, the whole checkpoint log record is treated as a continuous, unstructured record. These conventional tools search checkpoint log records for any occurrence of the search string or data value. Consequently, conventional search tools find matching data values, also known as “hits,” at various locations within a checkpoint log record. Unfortunately, these hits cross boundaries between log entries, boundaries within log entries, or occur at the wrong location within a log entry such that the hits are coincidental and of no use to the programmer. Such hits are false positives.
In addition, conventional search tools retrieve and present each checkpoint log record that includes at least one hit. Typically, this means that a high number of non-matching log entries are presented with the one or two log entries that contain the hit. Storing, printing, displaying, and sifting through the non-matching log entries together with the actual hit log entries can be tedious and labor intensive for programmers and system administrators concentrating on tracking down a software problem. The non-matching log entries make the results difficult to read. Furthermore, if the hit is a false positive, the receiving of these log records is wasted. In some instances, millions of lines of output are returned, the majority of which are extraneous.
From the foregoing discussion, it should be apparent that a need exists for a method for condensing reported checkpoint log data based on a query expression. The method should minimize false positives and the size of search results to ease storage requirements and log data analysis time.